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ABSTRACT 
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Using liicrocxjipiters for Assessment ai>d Error Arialysis 



CVenriew of Ocroutgr-Based Assessment and Error Analysis 
Numerous axithors have pointed out that the assessment and analysis of 
error patterns, v4ien carried out ccnsistently and cxxiscientiously, 
lead to inproved student learning. Given that this is true, 
we raost ask v*y student learning probleocs are not analyzed more 
frequently by classroan teachers? Bennett (1983) provides several 
possible reasons yAty error analysis is seldon carried out in the 
classroan. First, he suggests that too often teachers have 
not been trained to carry out the assesanent and error analysis 
process. Even if teachers want to analyze student error patterns 
they often do not have the necessary tedinical skills. Second, 
Bennett points out that the assessment and analysis of error 
patterns often can be a very labor intensive process, requiring a 
significant amount of time and effort. When asked, most 
educators report that they siirply do not have the time required 
to co n d u ct error analysis procedures fidthouc^ they knov error analysis 
is inportant. As a result, assessment and error analysis is a low 
priority activity in most dassrocns. 

Although thj>3 rooiograph should help you to develop the techniccil 
skills necessary for conducting error analyses, the problem of 
"too little time" is not as easily overoome. Recently, however, a number 
o£ aducators have begun to exploit the power of microccnputer technology to 
reduce tiie time required to assess and analyze student learning problens, 
Althou^ ccTfxiterHoased assessment is only in its infancy, it has already 
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shown potential for saving time as well as iaprcrving the accuracy of the 
error analysis process (Hasselbrlng, 1986) . 

The purpose of this monograph is to provide a general 
discussion of hcxi^ micro canA iters can be xosed in the assessntent 
and error ansdysis process. When e^ropriate, we will provide 
exacples of specific software programs that can be \ised for 
assessing and analyzing student performance. Ihrou^iout the 
m onograph^ we have classified ocnputer-^Dased assessment and error 
analysis sof^iware into five different categories; these inclixie: 
(a) test generation, (b) test scoring and reportir^, 
(c) interactive assessanent^ (d) curriculimHDased assessaoent, ard 
(e) e^qDert systems. A discussion of each of these software t^pes 
follows. 

Test Generation 
Often^ the first part of conducting an analysis of student 
errors involves the selection or development an appropriate 
assesanent instrument. VJhile the process of selecting an 
apprcpriate instrument can be tedious and time consuming, 
selecting an a ss e ss m ent instrument is not nearly as burdensoroe as 
developing one. Anya-^ *)o has gone through the process of 
developing an as se s s m ent instniment knows that it can take mar^ 
hours. Further, the problem is ccnpounded when parallel versions 
of an instrument are required. Recently, however, test 
generation programs have been developed that can be used to 
remove the tedium and reduce the time it takes to create 
assessment instruments. 



Using a test generaticn program, a teacher can quickly and 
easily create an assessment instrument that covers specific 
skills and ctojectives at a ^^ecified level of difficulty. 
Generally, these programs are designed to allo^ the teaciier to 
build banks of test items that are classified by topic, question 
format, or objective and then have the ounx i t e r select and print 
out ^)ecif ic assessment items based vqpon criteria provided by the 
teacher. 

such exanple of a test generation p r ogram is called 
Exam . Developed by the Brownstone Research Gxx/ap, Exam can be 
used to create assessment instruments as yaell as keep 
cdqputerized records of student performances on these tests. 
Before generating a test, the teacher must first use Exam to 
develop a bank of test items covering the topic or skills to be 
ass^sed. The test items can be written in a Vcuriety of formats 
^(*uch incli.de: multiple choice, true/false, matching, or essay. 
In addition to categorizing assessment items by type. Exam allows 
the teacher to categorize items by topic or level of difficulty. 
Once the test items are developed they can be stored on disk for 
future use. The stored item bank can be edited at any time by 
adding items, deleting items, or by dianging the vrording of any 
question. 

Following the creation of an item bank. Exam can be used to 
select specific items frcm the bank and print an assessment 
instrument. The teacher enters the criteria to be used for 
selecting items frcm the bank, such as difficulty level and type 



of itan, and the cxxqputer does the rest. A variety of 
instruments can be created using the same item bank by edtering 
the type of questions, the orcier of questions, or the difficult^/ 
level. This gives the teadier an opportunity to individualize 
the evaluation of students by creating and administering 
different versions of an assessment instrument that covers the 
same information. 

]^ allowing the ocnputer to assist in the development of 
assessment instruments, teachers can quickly and easily generate 
a series of tests. If these instruments are carefully sequenced 
with regard to item difficulty, the teaciier is able to determine 
at \*at point a child's skills break down or in vAiat ^)ecific 
area. Not only is this information useful for developing 
remediation progr a ms but this information is also liseful for 
documenting when and skills have been mastered. 

Test Scoring and Reporting 

Test generation programs lUce the one described above are 
useful in as lojcti as they allow teachers to create tests \rtuch 
serve as diagnostic tools. However, after generating the test 
and giving it to the student the teaciier uaast still score the 
student re^xanses and analyze the results. Currently, there are 
ccu y u t e r programs that can score and sunmarize student responses 
on an asses s m ent. The earliest of these progr am s were developed 
primarily for school psychologists to cQlow them to quickly score 
and sunmarize standardized tests such as the WISC-R, WAIS-R, 
FIAT, and WoodoocknJohnson Psydio-Bducational Battery. These 
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progrcxms allowed p^ydbologists to bo more productive since they 
no longer had to spend hours scoring and suninarizing the tests 
manuELlly. The cotputer was able to do nuch of this for them. 

When using a scoring program, the test is administered in 
its traditionEd. form viiidi is lasufiOly a p^Der and pencil format, 
or in seme cases orally, vdiich means the examiner most record the 
student's re^xaise^ In neither case does the student interact 
with the cctqputer, only the examiner. Ihe examiner most enter a 
student's scores froa the e^dministraticn of the test into the 
cccputer. This is generally done in one of two ways, either by 
typing the information throu^ the te^xard or by having the 
ccitputer read the information frm an optical scanning sheet. 
After entering the student's assessaiient information, the oonputer 
then sumnarizes and prints out the results in report form. Ihe 
primary advantage of this type of p r ogr am is that it saves time 
hy freeing the examiner fron such clerical tasks as arMing raw 
scores, loolung throu^ conversion tables, and in some cases 
writing reports. 

Recently, a number of scoring progr a ms have been developed 
specifically for classrcxm teachers. One exairple is the JK>- 
SOQRE Systems by Pro-Ed. In each of the FRO-SOQRE programs, the 
examiner first administers the test to the student in the 
traditionetl manner and then enters the student's re^)onses into 
the program. Ihe ccRputer then generates a nulti-^jage report 
that includes; raw scores, standard scores, percentiles, and 
descriptors for eacii subtest as well as a cognitive ^Jtitude 



score. Althou^ these programs do not pzxvide detailed error analyses, 
the information that is provided can be u^seful as part of the overall 
assessanent process. 

One progr a m designed ^^ecifically to score, analyze, and 
report student ^^elling errors is the Ccc{3uterlzed Test: of 
Spelling Errors (dSE) (Basselbring, 1984) . Using the CTSE, the teacher 
types in the student's ^^lings to a list of 40 preselected 
words, and the program analyzes the spellings and generates a 
report of the types of errors esdiibited by the students Ihe 
report that is generated from the program is divided into four 
sections. Ihe first section provicJes demographic data on the 
student and the testing session. The second section simply 
suzmarizes the nuniser of words that were ^)elled correctly and 
the number spelled incorrectly. Section three of the r^jort 
lists the student's ^)elling of all words, both correct and 
incorrect. lastly, section four provides an analysis of the 
types of words ^)elled, types of errors, and error tendencies 
e:diibited by the student. A sanple report frm the CISE is shown 
in Figure 1. 



Figure 1. Sanple R^rt cn the Ccnputerized Test of Spelling Errors 



I. Demographic Data 
Student's Name: Ben H. 
Birthdate: 06/18/80 
Grade: 2 
Examiner; TSH 
Elapsed Examinaticxi Time: 00: 30? 58 

II. Sunroiry 

NUniber of Words Correct: 
NUniber of Words Incorrect 

III. Student's Re^xsnses 
Words Spelled Correctly 

ARPOW 
ATE 
EULL 
BAKE 
AKHUE 
BORN 
DARK 
HER 

CUITING 
TflDGHING 
AM 
CRM 



Date of Examination: 04/18/88 
Chronological Age: 7-10 
School: Oak Hill 
Place of Examination: Oak Hill 



20 Percert of Worths Correct: 50% 

20 Percent of Words Incorrect: 50% 

Words Spelled Incorrectly 
ANKEL (ANKLE) 
FL^S (FLIES) 
MUmj (MEU3N) 
ASEJ> (ASLEEP) 
GUINT (GIANT) 
lATTER (lATER) 
JUMPPING (JUMPING) 
DURKED (CROPPED) 
POW (PAW) 
FEUD (FIEID) 
THOW CIHOUGH) 
HAMER (HAMMER) 



LDOKED 


CIRKER (CaRACKER) 


GIRL 


BAKS (BABIES) 


HDFED 


ALOCN (AICNE) 


PCNY 


EARIYES (PAHi-JJbS) 


WCXM 


OBAX (OBEz) 


mpca 


EA3HHK (EAKEHER) 


STREET 


PEICE (PIECE) 


BOXES 


AME (AIM) 
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Ocgputerized Test of Spelling Errors EveiLxiaticn Sunmary 



IV. Diagnostic Error Analysis 

A. Word Types Number Incorrect Percent Incorrect 



B. 



1. Regular 


1 out of 6 


16 


2. Predictable 


12 out of 20 


60 


3. Irregular 


7 out of 14 


50 


Error lypes 






VtMBlS 






1. Schwa vo^i^ sound 


1 out of 5 


20 


2. Short vo^el sound 


3 out of 10 


30 


3. Long vcMal sound 


0 out of 11 


0 


4. Dicnranhs and 






diphthcngs 


6 out of 11 


54 


5. Rrocntrolled 






voxels 


1 out of 10 


10 


Ocnscnants 






6. Initial position 


0 out of 24 


0 


7. Medial position 


1 out of 23 


4 


8. Final positicsn 


0 out of 12 


0 


9. Blends and 






digraphs 


4 out of 13 


30 
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Generalizations and Batttems 

10. Suffixes 2 out of 11 

11. Affixation rules 4 out of 5 

12. Orthographic 

patterns 3 out of 6 

13. Finsd E rule 1 out of 5 
Error Tendencies 

14« Qnissions NUniber of vrords with omissions = 8 

15. Insertions Nunter of words with insertions = 

16. Substitutions NUnfaer of words with substitutions 

17. Order Errors Nuntoer of words with order errors 

Total number of order err o r s 
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In samaxy, test scoring programs liJoB the ones descrilDed 
can be xiseful in that they reduce the amount of time associated 
with the scoring and error an2dysis process. However, on the 
negative side, these programs still require that the teacher 
enter the assessnent information into the ccnputer. In the next 
section, we will focus on programs that edlow testing to be done 
directly on the cocputer. 

Interactive Assessment 
In many respects the t^pes of programs discussed thus far do 
not take full cKivantage of the cocputer. An area of cccputer- 
based assessment that is gaining in popularity and does take 
advantage of many of tte powerful characteristics of the ccnputer 
is interactive assessment. Interactive assessment differs fraax 
the pr ogr a ms just descrihai in that the ccnputer plays the role 
of the examiner and carries out and analyzes the assessment 
data* The advantages of interactive assessment are obvious* For 
one, huge savings in examiner time can be accrtied. This is 
especially inportant if teaciiers want to monitor student progress 
on a regular basis or if the^'' want to monitor the progress of 
se^reral students. Also, in soDe cases, examiner bias, 
administration errors, scoring errors, arxi invalid or erroneous 
analyses and interpretations can be more ti^tly controlled or 
eliminated. However, on the negative side, with indzeractive 
assessment the teadier is rearnoved fron the assessment process. 
In sorae cases this can create a black-box phencroenon \*iere it may 
be unclear as to hew the cccputer came \jp with the ancilysis, 
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Nevertheless, we believe that interactive assessment programs can 
with very little effort provide extreraely loseful cJata to the 
classrocm teacher. As these programs becone more scphisticated 
and better designed, they will beccane an inportant part of the 
assessnent and error aneilysis process. 

For the purpose of describing this new and exciting form of 
ass^sanent, we hacve selected two interactive assessment p rograms 
in the area of matheaoatics. Althou^ these programs are math 
oriented, frcxa the descriptions you will see that they provide 
very different data for the classrocm teacher; they were 
selected for this reason. 
Qironcmgtric Analysi s of Math Strategies 

Today, many teachers and parents are content v4ien stixJents 
with learning handles^ can ccnpute answers to basic math facts 
using counting strategies (i.e., fingers and number lines) or 
electronic calculators. Hcwsver, research by Resnick (1983) 
suggests that these procedures can interfere with the leamirxf of 
hi^^ier level math skills such as itultiple digit a^idition and 
subtraction, long division, and fractions. Most cognitive 
scientists today believe that as basic math skills beoome more 
hii^ily practiced, their execution requires less cognitive 
processing capacity, or attention, ard the student beccroes 
fluent. Since all people have a limited cz^city for information 
prnoessing, not having to use part of this limited capacity for 
performing basic skills means that there is more ca^city left 
for executing hi<^ier-order processes. Thus, it appears that the 
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ability' to socxxed in hi^ier^-order processes is related direc±ly 
to the efficiency with vMch these 1ck«^ level skills are 
executed. 

Recent studies hacve shown that by isolating non-fluent nath 
facts and presiding individuadized cccpiter-nediated training and 
practice on these facts that 3ven students with learning problents 
can learn to retrieve the facts frxxi memory (Hasselbring, Coin, St 
Bransford, 1987) . A ccmpcnent of this training relies on the 
ability to identify each student's repertoire of flvient and 
non-fluent facts. 

The best way to determine if a fact is fluent or not is to 
record how long it takes a student to answer a problem. For 
exanple, loost adults answer basic facts under .6 seconds \diile 
seme students may take vqp to 10 seconds or more because of 
inefficient strategies they are using. Iftitil recently, it has 
been virtually iiipossible to record accurately the re^xDnse 
lat^encies for individual math facts outside of a laboratory 
setting. Rirther, it is impossible to tell from traditional 
paper and pencil forms of assessment vAiicii facts have been 
memorized and whicii have not. Thus, teachers have been denied 
this vadijable source of assessment data. However, by providing 
teachers with a tool that edlows tljem to easily and reliably 
assess and monitor response latencies they will be better able to 
provide students with the fliaency training that they need. 

To make the classroom assessment of fluency feasible, an 
interactive ccnfxiter program called C3VMS (Hasselbring & Coin, 
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1985) was developed. CMS is an acronym for the Chixnccetric 
Amlysis of I&th Strategies* C2M5 is designed to record and 
analyze student response latencies ;for all basic facts in the 
four cperaticns. Basic addition facts are defined as all facts 
frm OH) to 12+12. Basic subtraction facts are defined as all 
facts from CM) to 24*12 with the subtrahend eiLways being between 
0 and 12. MLatiplication includes 0x0 to 12x12 and division 0/0 
to 144/12 with the divisor always being between 0 and 12. 

Prior to begizining the actual assessment, the student is 
given several days of log^oariing practice on numbers. During 
the practice periods, the ocnputer records the response latency 
for each mmtoer betwaen 0 and 24. Ihe response times for each 
set of numbers are used in the data analysis to factor out^ motor 
re^)onse time frcm the actual carputation time. In otiier words, 
the time required to find and press the number key is subtracted 
frcm the tot2LL time required to solve the problem. 

Ihe testing of the basic facts is done interactively, that 
is, the student takes the assessnent on the ccnputer. The 
assesanents for each of the four operations are given 
independently. CA^B presents the problens to the student cn xhe 
video di^lay \jnit of the ocnputer. Qnoe the assessment begins, 
the stud en t is presented with a discrete set of facts one at a 
time. The student reqpcnds by typing the answer to the problean 
lising the number keys on the top row of the ocnputer kej^xDai-d and 
then presse s the space bar to have a new problem presented. The 
student's response is timed from the presentation of the problem 
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to the pressing of the nunter key. If & two or three digit 
answer is required, the timing stops with the pressing of the 
second or third lusnber key. Student response latencies are 
recorded to the nearest one hundredth of a second. 

CAMS provides the student with a tailored assessaooent. lhat 
is, C3^ decides vMch problens the student receives based xjpon 
responses to past problems. So, for exainple, if a student has 
had difficulty with probleoos frcn the six and seven tables in 
BUltiplication, (3VMS will not continue to give the student 
problenjs from these tables. CWe attenpts to predict the 
problems that the student is able to answer and avoid problems 
that are too difficult. Hxus CMS avoids being overly 
frustrating for the student yet provides extresnely rich data with 
regard to the student's level of fluency. TJpyn cccpletion of the 
assessment, CMC analyzes the data and provides a three-part 
printed sunmary vAiidi includes a: (a) chroncrastric aneLlysis, 
(b) descriptive analysis, and (c) fluency matrix. 

Bie first section of the CAMS Report is a chroncnetric 
analysis profile. By plotting a student's latency data a visual 
profile of the student's strategy for solving the basic facts 
beocroes e^jparent. Uie second section of the CMS R^rt provides 
descriptive data on the student's level of fluency. This analysis provides 
data on the number of fluent facts, total nuntoer of prcblests presented, 
the accuracy of responses^ and the nuntoer of correct, incorrect 
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resparses. The fined, section of the report is a matrix. Shaded cells of 
the matrix indicate flx^ent facts. Uhshaded cells indicate nan- 
flxjient facts. 

In sum, CAItS presides the teacher with extresaely detailed 
information concerning a student's level of mathematiced fluency 
at a very low cost in terms of teacher time and effort. With 
CAMS, the true power of the cccputer is exploited. Ihe 
assessment is presented interactively on the oocputer, the test 
items are tailored to the student based upon decisions that the 
ccqputer makes during the assessment process, scoring is done 
totally by the cccputer, and r^xarts are generated that allcw the 
teacher of quickly analyze the students level of performance. 
Error Analysis in Mathematical Contputation 

Althcu^ the inportanoe of fluency in basic facts cannot be 
stressed too heavily, teachers are cautioned against the 
ooncliision that if students have developed fluency in the basic 
Mth skills that they naturally will becooe better at hic^ier^ 
order ccmpitations. On the contrary, the developcent of the 
effortless recall of basic math facts sinply permits the learner 
to adlocate a greater proportion of attention resources to the 
hi^ier^level processes. Systematic instruction in the hi^ier- 
order skills is necessary if these students are to master these 
skills. Uuis, an important part of this instruction is the 
identification and analysis of faulty algorithms that stuients 
may develop. 

Normally, faulty EtLgorithms are analyzed by having a student 
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oonplete <x woricsheeu containing a variety of problens and for all 
incorrect answers try to determine the student missed the 
prcjblem. Often, the reasons are quite apparent, but at other 
times trying to determine \Aiere the student went wrong is time- 
consuming and diff iculr, 

CTie solution is to have the cccputer analyze student 
responses and determine tlw cause of any errors. One set of 
programs that provide this type of analysis in addition, 
subtraction, imiltiplication, and division is the Math Assistant 
series by Scholastic Inc, The Math Assistant series has several 
of the chauracteristics of ccnputer^Dased assessment programs 
described throughout this diapter. For example. Math Assistant 
can be used to create diagnostic tests at any level of difficulty 
ty entering the probleaoas manually or hatving the ocnputer generate 
them autanatically; students can take tests at the ocrputer or on 
printed tests at their desks; \*en tests are taken on the 
occ|xiter, student responses are saved autcroatically and can be 
analyzed imnediately by the ccnputer, but if students take the 
test on pe^^er the students' answers nust be typed into the 
conpiter manually for error analysis. 

Error reports can be printed cifter student test answers have 
been e^itered into the ocnputer either by test-ta3cing on the 
ccRfuter or by entering them manually. Either individusLL or 
grtx^ error reports are available. An exaitple of an individual 
report is shown in Figure 2. 
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Figure 2. Sanple Individual Report on Math Assistant Individual Reports 



Individucil Report A Subtracticn 



This is a recx>rd of student errors cn all tests. Bie error number is 
listed for every incxsrrect answer. 'C shows a cxarrect answer. marks 
vAien no answer was given. • # ' means that no answer was required (because 
the test contained fewer than 20 prcblenis) . 

FCOBIZM NUMBER 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 



NftME: JftMES ROSE TEST: LEVEL WE/ SUB SCORE: 

50% 

16 110CCC112CC######### 

# 
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Individual Report B Subtraction 



This report sSxms the total number of times (in parentheses) each error was 
made cn cill tests. 

Error 1 JftMES EOSE ( 3) 

Error 3 JftMES RCSE ( 1) 

Error 10 JM',^ POSE ( 1) 

Error 12 JftMES ROSE { 1) 

Error 16 JftMES POSE ( 1) 
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Individixal Reports prwide a suzmary of the number and 
percent of problens answered oorrectly, vMch problems were 
answered correctly, and the type of error made on prcfclems 
answered inoorrectly, ttifortunately, in order to see the 
problems that the student Tnifyyyj the test nust be printed out. 
Also, the error type is identified only by a nunfcer. In order to 
identify the error type one mast go to the inanual for a 
descriptor. For exaziple, Error 1 is r^resents a regroi^jing 
problem vAiile Error 10 represents an error on basic facts. 
Further, \*en itultiple errors occur en a single problem, only a 
single occurrence shc^ on R^xsrt A. Double errors are shown 
only on R^»rt B. 

Grocp R^»rts are virtually identicsLL to IndividU2d Reports 
except that error data are reported for nultiple students. An 
exanple of a Grotqp Report is shown in Figure 3. As can be seen, 
type 10 errors were made by four of the seven students. With 
this infonnation, teacdiers can prioritize their instmctional 
objectives for groap and individual teadiing formats. 

In sum. Math Assistant provides a loseful error analysis of 
student cooputation err o r s in the four operations. Perhaps Math 
Assistant and other similar programs can be used most 
productively by having students take tests on a regular basis so 
that errors can be monitored on over time to determine if the 
instructioncil progr am is effective. Since it takes only a few 
minutes to prq^are a test using Math Assistant it provides an 
easy way to regularly monitor student performance. Ideally, 
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these performance data should be iised in oonjxmction with 
curriculum-based assessment software described in the next 
section. In this way, student progress can be monitored in a 
systeraatic manner. 

CHrriculum-Based Assessment 
In the past two dncades, ^)ecial education teachers have 
been trained to collect daily performance data to monitor the 
progress of their students. For exanple, consider an 
instructioraLL pr ogr a m designed to teach a student place-value 
subtraction. Each day the teacher instructs on the rules of 
subtraction and then probes the student cai 40 problems viiich test 
the concepts of place-value subtraction. On each probe the 
teacher records the amount of time it takes the student to 
occplete the 40 problesns and reconis the number of correct and 
incorrect re^»nses. 

Ihe purpose of collecting dsdly data is to enable the 
teacher to determine whether the instrtactionsLL progr a m is workir^ 
as planned. Specific techniques have been developed to enable 
tea ch e r s to use this type of classrocm data to determine \*yen an 
instructional strategy should be changed. Haring, Liberty, and 
VJhite (1980) developed a set of guidelines, called data-based 
decision rules, to help teaciiers determine not only when 
ixistructional strategy should be chax^ed, but also what kind of 
change is likely to produce favorable results for a particular 
student at a particular time. The decision rules help the teacher 
choose the strategy that has the hi^buest probability of success. 
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Figure 3. Sanple Grot?) Report on Math Assistant 



Grcxqp T^eport A Subtraction 



This is a recx^rd of student errors on all tests, ihe error nux±>er is 
listed for every inoorrect answer. shows a correct answer. marks 

\A)Br\ no ansv^ was given. means that no ansv^er was required (because 
the test contained fewer than 20 problesns) . 



ITOHLfM NUMBER 



NAME: BENJAMIN LEE TEST: IJEVEL CtlE/ SUB SCORE: 

100% 

cccccccccc######### 

# 

NAME: JAMES ROSE TEST: LEVEL OtlE/ SUB SPORE: 

50% 

16 110CCC112CC######### 
# 
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NAME: amn PABSCNS TEST: LEVEL GtfE/ SUB SCCKE: 

70* 

2 10 15CCCCCCC######### 

# 

NMffi: BENIftMIN I£E TEST: lEVEL TWO/ SUB SGCKE: 

90% 

10CCCCCCCCC######### 

# 

NWffi: CmSCl PARSONS TEST: lEVEL TV3D/ SUB SOORE: 

60% 

21513 10CCCCCC######### 

# 

NAME: SAMOAL JACKSCN TEST: lEVEL THREE/ SUB SOORE: 

90% 

ioccccccccc#####-#### 

# 

NAME: BE30AMIN I£E TEST: LEVEL TSJBEE/ SUB SOORE: 

70% 

1 10 CCCCCCC 10 ######### 

# 
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Grot?) R^rt B 



Subtraction 



Uiis r^xDrt shows the totad nu^iDer of times (in parentheses) each error was 
made on all tests. 



Error 


1 


JAMES ROSE 


( 3) 






HtHJAMIN 


( 1) 


Error 


2 


CaiHY PARSONS 


( 2) 


Error 


3 


JAME3 ROSE 


( 1) 


Error 


10 


JT^MES ROSE 


( 1) 






C3mff PARSONS 


( 2) 






BEtDMGN T3?R 


( 3) 






SAMUAL JACKSai 


( 1) 


Error 


12 


JAMES ROSE 


( 1) 


Error 


13 


amn parsojs 


( 1) 


Error 


15 


CAUff PARSONS 


( 2) 


Error 


16 


JAMES ROSE 


( 1) 



24 

O J 



In order to xjse these decision rules in the cxDnventioned 
manner, the teac±yer plots the data cn semi-logarithniic grc^ii 
paper. Ihe initied three days of data are plotted as a baseline, 
and an "adm star'* is drawn at the intersection of the desired 
level of performance and the target date for achieving that level 
of performance. A •♦miniiaum oeleraticn line" is then drawn froa 
the mi(^»int of the baseline data to the aim star. This line 
indicates the mininiim level of acceleration or deceleration 
necessciry to achieve the criterion level of performance by the 
target date. 

As the teacher continues to conduct the instructional 
pr ogr a m, she collects and charts data. Ihree consecutive days of 
data falling below the •celeration line indicate that the student 
is not learning satisfactorily and that a change should be made 
in the instructioned strategy. In addition, a "line of progress" 
is drawn between the median of the most recent three days of data 
and the median of th3 three previous days of data to determine 
the trend of the student's performance. If the stuJent is not 
progressing and/or has fedlen below the ^oeleraticn lir^S and 
additional flow chart is used to determine vAiat type of cdiange in 
the instruction£d strategy is most likely tc be successful. 

Research on the effectiveness of curriculum-based c^sessiasint 
has indicated that this methodology is quite prcmisirg for 
irproving student acJiieveoaent. For exaitple, Fuchs and Flichs 
(1986) analyzed 21 researdti studies that evali^ated CEA 
proced u res. The results of this aneilysis indicated the use of 
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CBk prooedures significantly increased the acadaodc achievenent 
of students vdtxDse teachers lised these procedures > Frcn these 
findings, we can assume that vdien a student's instnx::tiQnal 
programs are monitored lasiiig CBA procedures they will achieve 
loxli more than students \A)OGe progr a ms ar^ not monitored using a 
CBA. eqpproach* 

Rk±s and Fuchs conclude that the usc^ of CEA procedures 
increases student acadenic achievement and that the greatest 
gains can be expected vAien teachers use specific data-based rules 
and graph performance data for making instructional decisions. 
Despite the apparent effectiveness of CBA procedures, cill indica- 
tions are that teaciiers are reluctant to enplay them. In a 
national survey of ID teachers. Wesson, King, and Deno (1984) 
fcund that, adthcu^ teadiers believe that CBA procedures are 
effective, th^ do not use the methodology because it is too tiine 
consuming. 

Oaqputer^Based Monitoring and Decision Making 
In an attenpt to make curriculimH^ased assessment less time 
consuming and easier for teachers to inplesnent, a number of 
developers have proposed the lose of microcoiputers for 
iirplementing CBA procedures (Fuciis, Fuchs, Hamlett, & 
Hasselbring, 1987; Fuchs, Deno, & Mirkin, 1984; Hasselbriixf & 
Hamlett, 1984; West, Young, & Johnson, 1984). Basically, these 
itcnitoring programs have been designed to assist teadiers in 
storing, graphing, and aned^^zing student performance data. One 
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such ocnputer program that has been xosed sixxsessfully in ^)ecial 
edu::ation is AIM5TAR. 

AIMSTAR is an integrated set of occputer programs that are 
designed assist teachers in storing, graphing, and analyzing 
student performanoe data. To i2se AIM5TAR, the teacher creates a 
student data file. Descriptive infomaticn about the student *s 
instnictioned. program, the program objectives, and teaching 
procedures are inolixied. Following each teaching session, the 
teacher enters student performanoe data into the occfxiter. For 
exaitple, the number of correct and incorrect respcanses ^diibited 
by the student and the amo u n t of time required for the student to 
ocnplete the trialsis entered. AIMSTAR then stores this information and 
aillows tlie teacher to grs^ the student's data, £qpply data-based 
decision rules, and produce a printout giving the status of the 
student's instructional program with recccioended changes \ighen 
exp ropr iate. The ^5kdvantage of this type of analysis and report is that it 
is ininediately obvious ;*en a student is having dif f ic?ilty with a skill anl 
when an instnictional strategy is or is not working. 

It should be es(3hasized that monitoring and decision making 
programs, such as AIMSTftR, do not eliminate the need for teacher 
intuition and judgment in planning instruction. Rather, these 
programs si^plement teacher jxxJgment by prcjviding adiitional 
eaoppiricaLL data and ajietLytic procedures. Teachers using this 
technology are able to reqsond more flexibly and effectively to 
changing student needs and to produce greater student growth. 
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Expert Systaos: The Future of Oonputer^Based Assessment? 
Ihus far, we have discussed assessment and error analysis 
programs that are used priinarily to reduce the tedixm associated 
with the analysis of student learning problesos* For the most 
part, however, these p rogr a ms have in no way provided the same 
intelligent insi^it into the assessment and error ansdysis 
process that human diagnosticians can prwide* But yA)at does the 
future of cocputezHxised assessment and err o r analysis hold? 
Consider the following scenario: A teacher in a rural area is 
perplexed by a learning problem esdiibited by cne of his students^ 
Ihe school is too small to have the regular services of a school 
psychologist or diagnostician^ However, with this child, the 
teacher feels that he needs an expert's cidvioe to help with an 
analysis of tlie student's problem and to reocomend an effective 
instructional progr am ^ So he goes to the ccnputer at his desk 
and enters into a dialogue with it. Ihrouc^ the program he is 
using ha has access to the knowledge, jix3gment, and intuition of 
the country's best edLkoationed diagnosticians. Ihe progr am 
queries the teacher CGnoeming the student's problems; requesting 
inf ormaticn fron the teacher that will help the system ccme up 
with an analysis and prescription that will have a hi^ 
probability for success. In short, the ocnputer serves the role 
of an expert consultant that the teacdier can call vpcsn at any 
time and disaiss problems that students are having in the 
classrocm. 
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Iftifortunately, the gap betMaen the scenario abcr/e and 
reality is large. Nevertheless^ both ixmju ter and cognitive 
scientists are itaking great strides in proving Intelligent 
consultants cad^led e^^pert systeDos. Expert systesns have evolved 
over the past 25 years frcm the field of artificial intelligence 
and can be defined as cccputer progr a ms ca^^able of reaching a 
level of performance ccnparable to that of a human e:q)ert in scrae 
specialized prablen (tana in (Nau, 1983) • Expert systecis are 
unlilce conventiored application pro g rams in that they are the 
first systesns designed to help humans solve ccnplex problems in a 
cxaiiuuA sense way. These ^rstaons i^se the methods and information 
aoguired and developed by a human esq^ert to solve problems, make 
predicticns, suggest possible treatments, and offer advice that 
is as accurate as its human counterpart. 

Vlhile expert systems in the field of education are only in 
the early phases of developoent, expert systems are being used 
more widely in other fields. Sane of the e:q)ert systems 
currently in use include: Gaduceus, \Aiicii helps doctors diagnose 
m edi c al problems; CATCH, vAiich can scan 250,000 photographs to 
assist New York City police in identifying criminal suspects; and 
Pro^jector, vfeLcii sifts geological data to predict the location 
of mineral deposits. Ihe practiced c^licaticns for expert 
systems abound. VJhenever human experts are in great demand and 
short supply, a cotputerHDased consultant can help to aiiplify and 
disseminate the needed expertise. 
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Expert systems are designed to perform the sane role as the 
human expert consultant, that is, provide advice in situations 
vAierB hi^ily specific }ai0vl6dge and e^qperienx is n eeded. 
Ideally, an expert system should provide the user with the same 
inf ormaticn and play the same role as a human expert ^f^tysn placed 
in the same situation. For exanple, a medically-oriented expert 
systan might guide an inexperienced intern by asking relevant 
questions about the case. Hie intern would respacd by l^ing the 
ansv^ers to these questions. The intern continues this dialogue 
with the cGcputer until the expert system has sufficient data on 
whicJi to make a diagnosis. At this point, the intern either 
accepts the diagnosis or siiiply uses the ccnputer's diagnosis as 
another piece of data since these systems are intended to 
corplement, not replace, a physician's jix^gment and intuition. 

Hie development of an expert system is an extremely 
laborious process. It requires that a human expert (or grot^ of 
experts) in some dcmain is willing to spend hundreds of hours 
€3q)laining to a "knowledge engin^r*' how s/he solves particular 
problems. It is the re^xansibility of the "Tcnowledge engineer*' 
to develop a knowledge base and set of decision rules that 
r^^jresent the thinking process of the expert. 

Current expert systems tedinology seems best suited to 
diagnosis or classification problems vdiose solutions depend 
primarily on the possession of a large amount of qycialized, 
factual, and enpirical kncwledge (Duda & Shortliffe, 1983) . 
Thus, it is only logical that expert s y stems be developed in 
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educaticn for assessing and diagnosing error patterns and 
learning probleiDS. Althou^ Math Assistant , discaissfyj earlier in 
this chapter, was designed to provide the teaci^er with expert- 
like informatidi, by definition. Math Assistant is not a true 
esq^ert systesn« 

To date, severed expert systems have been developed that are 
designed to assist in the diagnosis and analysis of student 
learning problesns. One such p ro t ot y pic system developed by 
Oolboum and Mnlipod (1983) assists teachers in the diagnosis and 
analysis of reeviing problesns. Ihis system guides the teacher 
through reading diagnosis frcm the initial su^icion that a 
reading problem exists to the point at viiich sufficient 
inf ormaticn has been gathered to plan an appropriate remedial 
program. 

With this system, a dialogue is oonducted betu^een the vser 
and the cxuputer, with the system posing questions or making 
e^ropriate su^estions. If desired information is not available 
the system provides the diagnostician the option of stopping the 
dialogue in order to obtain the needed data. In sane cases viiere 
it is inpossible to obtain the desired information the system is 
capable of handling inooiplete data. However, in the case v*ien 
the system most have further inpat in order to continue the 
diagnosis, the system reiterates What data are required and then 
terminates the session. 

Diis expert system does not test the student directly, nor 
does it manage the testing activities. Instead, the teacher or 
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diagnostician perforins the tasks su^ested by the system (such 
as adxnixustering a specific test) and enters this information 
into the sustm. After these new data have been entered, the 
systan analyzes thii. information and proposes the next step in 
the assessment process^ Vlhen a sufficient amount of information 
has been gathered and entered, the system prcrvides a report of 
its diagnostic findings. Ihe teacher can then plan a renedicil 
pi :Kxp:e m based on these results. An obvious extension of this 
system is to have it prescribe appropriate remedial strategies 
and instructional techniques based \spon the diaignostic findings. 

performance of this expert dis^inostic systera has been 
evaluated by cccparing it against human diagrx^sticians. When 
subjected to a number of test cases it was found that the e>^)ert 
system^s diagnostic reports were consistently good. In con tr ast, 
the diagnostic reports prepared by the human experts varied 
dramatically in terms of style, format, rRfviahility, relevance, 
and accuracy. Of course, sane of the r^xDrts of the human 
diagnosticians were judged better than the r^xDrts of the expert 
system while many others were judged as inferior. 

Other exairples of e)q)ert systems capable of diagnosing and 
analyzing student learning problems exist and others are 
currently vmder development. DEBOGCy; one of the earliest expert 
systems designed for education (Burton, 1982) , is used for 
diag n osing student errors or '*bugs" in the domain of place-value 
subtraction. Ihe system is based on a model vdiere student errors 
are not seen as randcm but rather as predictable bugs in an 
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algorithm. After analyzing student errors in subtraction the 
e3^)ert system attespts to determine the student's '*buggy** rules 
for solving the problem. DEDSUGGY does this by hypothesizing the 
student's bugs; then it attempts to predict not only vA^ether the 
student will get a siinilar probleoEi incorrect but edso vAiat the 
exact answer will be. This system has been vised quite 
successfully for diagnosing faull^ algorithms ooimc nly used by 
students when solving subtraction problems. 

It appears that the developoent of expert systems for 
assessing learning problems shc^ great premise. Hcm^ever, we 
Bust view the energence of es^^ert systems with mixed esaotion. If 
expert systems are developed with pedagogical soundness then 
these systems will no doubt benefit teachers and stucknts. On 
the other hand, it is very likely that a number of systems that 
are pedagogically unsound will find their way to the educational 
marketplace. Accordingly, wb must be cautious and evzduate very 
carefully an/ conputer pr ogr am that proposes to diagnose and 
ansLLyze student learning problems. 
Sungnarv 

OoBputer^-based assessment and error analysis, while oiily 
beginning to be used in the schools, appears to offer great 
promise for cfvertxndng many of the problems associated with the 
assessment of student's learning difficulties. When progranroed 
appropriately, microoonputers can remove nucli of the tedium 
associated with the administration and scoring of dic^nostic 
instrtsnents. with assessment programs becoming more 
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sophisticated/ the <.xii|x iter is playing a larger role in the 
analysis of learning probleaos. For exanple, in the case of the 
CMS program described earlier in this chapter, the cccfxiter 
presents the student with e^^propriate assessment items, mcTiitors 
stude.'Tt re^xanses, scores, sunnarizes, and prints out a report en 
the s tuden t's performance. CAMS and other analysis programs are 
relatively sinple in occpariscn to ^he e^qpert systems being 
developed currently, nevertheless, they have been shown to be 
extremely successful for satving examiner time, reducing scoring 
errors, and providing teachers with information that would be 
difficult to obtain without the xase of a ocnputer. 

The assessment and analysis p rogr a ms currently in use are 
only precursors to the more elaborate and powerful intelligent 
systems that will be aivEOlable in the r^ext five to ten years. 
These systems will be able to guide the teacher throi^ the 
ne cessary steps for the analysis of learning problems, test the 
student directly \(Aere ^appropriate, analyze student performance 
data, and prescribe appropriate instructional strategies for 
remediating the student's problems. Althoi^ it is unlikely that 
the use of conputers will eliminate aH of the difficulties 
involved in assessing and analyzing learning probleois, existing 
research su^ests that the process can be enharjced throu^ the 
responsible use of this technDlogy. 
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Glossary of Tersss 

Aim Star: Ihe point plotted at the intersection of the desired 

level of perfonnanoe and the target date for achieving that level. 

Alogrithm: A step by step procedure for solving an arithmetic 
problem* 

Achievement test: An instrument xised to assess the amount or level of 

learning that a student has acquired* 
Arithmetic operaticn: A solution to the ocnputation of a prcblCTi 

(e.g., addition, subtraction, nultiplication, etc.)* 
ArtifiaiAL intelligence: A branch of ooiputer science devoted to 

researciiing ways to make the ccnputer eculate those attributes we 

classify as human intelligence. 
Autcinaticity: Ihe imnediacy of a stLrient respcansa to a stisultis (e«g. , an 

arithmetic problem) . 
Bias: Uhfadmess in testing \Aiich can include interpretation of scores, 

test content, the development process and the procedures with which 

the test is aidministered. 
GAMS (Chrcncmetric Analysis of Math Strategies) : An interactive c ai p u ter 

program designed for classroom assessment of fluency in mathematics 

facts. 

Ceiling: Ihe point in test administration \iAiere the student receives no 

credit for eCLl subsequent/more difficult items. 
Ciironcmetric analysis: An analysis of a student's response latency data 

vJiich provides a profile of the student's strategy for solving the 

problems. 
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Copy: To Imitate or re pr oc k io e an original itan or model. 
Critericn-referenced test: An assessment device designed to measure 

the student's ability to reach a designated level of performance 

on a ^5ecdf ic task. 
Curriculum-based measuresoent: An assessanent system within \^ch testing 

mRtPirials are derived fronm curriculum content, measurement is 

ongoing, and assessment information is used to develop instructional 

programs* 

VEBJGGi: One of the earliest expert systems designed for educaticn vised 
for diagnosing student errors or "bugs" in the dcmain of place- 
value subtraction. 

Diagnostic assessment: Fonzal and informal tests used to ocnpare 

achievement levels, to sort people into groups, to identify skill 
deficits and to measure growth. 

Diagnostic test: A measuring device used to determine causes of learning 
problems and/or specific strengths or weaknesses. 

Error analysis: Ihe individual analysis of a work sanple to identify 
specific error types and set priorities for teaching. 

Expert system: Ocqputer software that enulates the functioning of a 
human expert in a particular field of knowledge. 

Faulty algorithms: A skill or series of skills a stix3ent incorrectly has 
developed to solve ^^ecific types of problems. 

Formative test: An assignment/test administered v*iile learning is in 
progress. 
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mdividualized education plan (lEP) : A written accxxmt of ed u ca t ional 

objectives, strategies, aarriculum modifications, and classroom 

aoocnodations for a diild with learning and/or behavior problems; 

required by EL 94-142 • 
Intercictive assessment: Software programs in \i4iich the ax|Aiter acts in 

the role of examiner carrying out and analyzing the assessapaent data. 
Item analysis: Uie process of collecting, suxnnaryizing and lasing 

information about individual test items, especially infornation about 

perils' responses to items. 
Kxjwledge engineer: Person that develops a knowledge base and set 

of decision rules that r^resent the thinking process of a human 

ecjqpert or set of expsrts in a particular domain. 
Line of progress: The trend of a student's progress drawn from the median 

of the three most recent days of data and the median of the three 

previous days of data. 
Minimm 'oeleration line: A line drawn frcm the student's current level of 

performance to the desired performance level (aim star) indicaicing 

the miniinmi level of cKxaleraticn or deceleration in the student's 

performance necessary to adiieve the critrion level of performance. 
Norm-referenced test: A valuative instrtiment in v^ch a person's 

performance is judged in carparision with the average performance 

of others in a large reference grot?) of similar age and/or 

educationetLlevels . 
Sunsaative test: A test administered at the end of a diapter or 

unit of instruction. 
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Test generation programs: Software pr txjx' aTnR designed to 2tLlaw teachers to 
quickly and easily create an assesssnent instrument that oovers 
specific skills and objectives at a given level of difficulty. 

Test scaring and reporting programs: SoftWcire prograins designed to 

score emd sumarize student re^x:nses en an assessment instrument. 
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